ENCODER AND DECODER

Leveraging Pre-trained Checkpoints for Sequence Generation Tasks (2020.tacl-1)

Challenge:	Neural machine translation (NMT) uses sequence to sequence architectures, but requires a huge amount of parallel data.
Approach:	They use Elastic Weight Consolidation to regularize weights of two language models . they then fine-tune the model on parallel data to avoid forgetting the original task .
Outcome:	The proposed method achieves BLEU scores similar to the previous work, but is slower and requires less training data.

Copied to clipboard

Sascha Rothe, Shashi Narayan, Aliaksei Severyn

Language-Aware Multilingual Machine Translation with Self-Supervised Learning (2023.findings-eacl)

Challenge:	Unsupervised pre-training of large neural models has revolutionized Natural Language Processing.
Approach:	They propose to use pre-trained checkpoints for Sequence Generation to initialize a Transformer-based sequence-to-sequence model that is compatible with these checkpoint.
Outcome:	The proposed model is compatible with pre-trained BERT, GPT-2, and RoBERTa checkpoints and achieves state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentance Fusion.

Copied to clipboard

Haoran Xu, Jean Maillard, Vedanuj Goswami

Responsible & Trustworthy NLP

Challenge:	Multilingual machine translation (MMT) is a challenging multitask optimization problem because of lack of a framework to learn language-specific parameters.
Approach:	They propose a self-supervised learning task that denies monolingual data to MMT . they then propose 'intra-distillation' task that co-trains with MMT task .
Outcome:	The proposed approach outperforms three state-of-the-art methods on 8-language and 15-language benchmarks.

What Works and Doesn’t Work, A Deep Decoder for Neural Machine Translation (2022.findings-acl)

Copied to clipboard

Zuchao Li, Yiran Wang, Masao Utiyama, Eiichiro Sumita, Hai Zhao, Taro Watanabe

Improving the Transformer Translation Model with Document-Level Context (D18-1)

Challenge:	Deep learning has demonstrated performance advantages in a wide range of natural language processing tasks.
Approach:	They propose to deepen the decoder layer in a Transformer model to reduce the difficulty of deep learning.
Outcome:	The proposed method can deepen the model on both the encoder and decoder at the same time, resulting in a deeper model and improved performance.

Copied to clipboard

Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, Yang Liu

Pretrained Bidirectional Distillation for Machine Translation (2023.acl-long)

Challenge:	Existing models for document-level context translation ignore documentlevel context.
Approach:	They propose a document-level context encoder to represent document- level context and integrate it into the Transformer model.
Outcome:	Experiments on NIST Chinese-English and IWSLT French-English datasets show that the proposed translation model outperforms the Transformer model significantly.

Copied to clipboard

Yimeng Zhuang, Mei Tu

Explicit Cross-lingual Pre-training for Unsupervised Machine Translation (D19-1)

Challenge:	Existing studies have focused on language knowledge transfer from pretrained models to neural machine translation models.
Approach:	They propose to use masked language pretraining to efficiently transfer bidirectional language knowledge to NMT models.
Outcome:	The proposed method can significantly improve machine translation performance and achieve competitive or even better results than previous methods.

Copied to clipboard

Shuo Ren, Yu Wu, Shujie Liu, Ming Zhou, Shuai Ma

Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languages (D19-1)

Challenge:	Existing approaches to build initial unsupervised machine translation models with cross-lingual n-gram embeddings are inexplicit and limited.
Approach:	They propose a cross-lingual pre-training method that incorporates cross-linguistic training signals into pre-trained models by randomly choosing source n-grams in the input text stream.
Outcome:	The proposed method significantly improves the performance of unsupervised machine translation.

Copied to clipboard

Yunsu Kim, Petre Petrov, Pavel Petrushkov, Shahram Khadivi, Hermann Ney

M3: A Multi-View Fusion and Multi-Decoding Network for Multi-Document Reading Comprehension (2022.emnlp-main)

Challenge:	Using parallel corpora, we train a single, direct NMT model for non-English language pairs.
Approach:	They propose three ways to increase the relation among source, pivot, and target languages in pre-training . they use additional adapter component to smoothly connect pre-trained encoder and decoder .
Outcome:	The proposed methods outperform multilingual models up to +2.6% BLEU in WMT 2019 French-German and German-Czech tasks.

Copied to clipboard

Liang Wen, Houfeng Wang, Yingwei Luo, Xiaolin Wang

Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization (2023.findings-emnlp)

Challenge:	Existing methods for multi-document reading comprehension cannot make full of the advantages of both approaches.
Approach:	They propose a multi-view fusion and multi-decoding method that integrates multiple documents for answering questions.
Outcome:	The proposed method improves on two mainstream multi-document reading comprehension datasets.

Copied to clipboard

Lei Lin, Shuangtao Li, Yafang Zheng, Biao Fu, Shan Liu, Yidong Chen, Xiaodong Shi

A Compact and Language-Sensitive Multilingual Translation Method (P19-1)

Challenge:	Recent studies show that sequence-to-sequence (seq2sequ) models struggle with compositional generalization (CG) a crucial property of human language learning is its compositional globalization (GC), the algebraic ability to understand and produce a potentially infinite number of novel combinations from known components.
Approach:	They propose a sequence-to-sequence (seq2sequ) extension which learns to compose representations of different encoder layers dynamically for different tasks.
Outcome:	The proposed model achieves competitive results on two comprehensive and realistic benchmarks, which empirically demonstrates the effectiveness of the proposed model.

Copied to clipboard

Yining Wang, Long Zhou, Jiajun Zhang, Feifei Zhai, Jingfang Xu, Chengqing Zong

Dense Information Flow for Neural Machine Translation (N18-1)

Challenge:	Existing paradigms for multilingual neural machine translation do not make full use of language commonality and parameter sharing.
Approach:	They propose a multilingual neural machine translation paradigm with one encoder-decoder model that makes full use of language commonality and parameter sharing.
Outcome:	The proposed method outperforms strong standard multilingual translation systems on WMT and IWSLT datasets.

Copied to clipboard

Yanyao Shen, Xu Tan, Di He, Tao Qin, Tie-Yan Liu

Tagging-Assisted Generation Model with Encoder and Decoder Supervision for Aspect Sentiment Triplet Extraction (2023.emnlp-main)

Challenge:	Recent advances in deep neural networks have improved learning performance for NMT . Residual connections allow features from previous layers to be accumulated to the next layer easily.
Approach:	They propose a densely connected NMT architecture that can train more efficiently for NMT.
Outcome:	The proposed architecture improves learning performance and attention quality on multiple datasets.

Copied to clipboard

Luo Xianlong, Meng Yang, Yihao Wang

HintedBT: Augmenting Back-Translation with Quality and Transliteration Hints (2021.emnlp-main)

Challenge:	Recent advances in the ASTE task have been driven by Natural Language Generation-based approaches, but most NLG methods overlook the supervision of the encoder-decoder hidden representations and fail to fully utilize the semantic information provided by the labels.
Approach:	They propose a tagging-assisted generation model with encoder and decoder supervision that enhances the supervision of the encoder-decoder through multiple-perspective tabbing assistance and label semantic representations.
Outcome:	The proposed model enhances the supervision of the encoder and decoder through multiple-perspective tagging assistance and label semantic representations.

Copied to clipboard

Sahana Ramnath, Melvin Johnson, Abhirut Gupta, Aravindan Raghuveer

Towards Robust Neural Machine Translation (P18-1)

Challenge:	HintedBT provides hints (as source tags on the encoder) about the quality of each source-target pair.
Approach:	They propose a method which provides hints to the encoder and decoder to improve the quality of BT data by providing hints about the quality.
Outcome:	The proposed method improves translation quality and performance in three low/medium-resource language pairs.

Copied to clipboard

Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, Yang Liu

Text Generation

Challenge:	Small perturbations in the input can severely distort intermediate representations and thus impact translation quality of neural machine translation models.
Approach:	They propose adversarial stability training to make encoder and decoder robust to perturbations by enabling them to behave similarly for the original input and its perturbed counterpart.
Outcome:	The proposed approach improves translation quality and robustness over strong models on Chinese-English, English-German and English-French translation tasks.

Sparse and Decorrelated Representations for Stable Zero-shot NMT (2020.findings-emnlp)

Copied to clipboard

Bokyung Son, Sungwon Lyu

Towards Enhancing Faithfulness for Neural Machine Translation (2020.emnlp-main)

Challenge:	Using a single encoder and decoder for all directions is a popular scheme for multilingual NMT.
Approach:	They propose a scheme that uses a single encoder and decoder for all directions . they show that enforcing sparsity and decorrelation on encoder intermediate representations mitigates this problem .
Outcome:	The proposed model degenerates when decoding non-English texts into English regardless of the target specifier token.

Copied to clipboard

Rongxiang Weng, Heng Yu, Xiangpeng Wei, Weihua Luo

Text Generation

Challenge:	Neural machine translation (NMT) has achieved great success due to the ability to generate high-quality sentences.
Approach:	They propose a training strategy with a multi-task learning paradigm to build a faithfulness enhanced NMT model.
Outcome:	The proposed model can generate high-quality sentences that are very close to natural language.

Encoder and Decoder, Not One Less for Pre-trained Language Model Sponsored NMT (2023.findings-acl)

Copied to clipboard

Sufeng Duan, Hai Zhao

Bridging the Structural Gap Between Encoding and Decoding for Data-To-Text Generation (2020.acl-main)

Challenge:	Existing methods for neural machine translation (NMT) use encoder-only enhancement or rely on specific multilingual PLMs.
Approach:	They propose a monolingual PLM-sponsored NMT model that lets both encoder and decoder enjoy PLM enhancement.
Outcome:	The proposed model improves on IWSLT14 En-De, De-En, WMT14 En -De and En-Fr tasks and achieves new state-of-the-art.

Copied to clipboard

Chao Zhao, Marilyn Walker, Snigdha Chaturvedi

Text Generation

Challenge:	Current sequence-to-sequence models require serialized input, resulting in loss of structural information.
Approach:	They propose a dual encoding model that incorporates the graph structure and caters to the linear structure of the output text.
Outcome:	Empirical results show that dual encoding can improve the quality of natural language descriptions.

Abstractive Multi-Document Summarization via Joint Learning with Single-Document Summarization (2020.findings-emnlp)

Copied to clipboard

Hanqi Jin, Xiaojun Wan

Information Extraction & Text Mining

Challenge:	Existing methods for document summarization are extractive and abstractive.
Approach:	They propose to jointly learn an abstractive single-document decoder and a decoding controller to aggregate the decoded outputs for multiple input documents.
Outcome:	The proposed model outperforms several baselines on two multi-document summarization datasets and proves that it is useful for both tasks.

KFCNet: Knowledge Filtering and Contrastive Learning for Generative Commonsense Reasoning (2021.findings-emnlp)

Copied to clipboard

Haonan Li, Yeyun Gong, Jian Jiao, Ruofei Zhang, Timothy Baldwin, Nan Duan

Solving Math Word Problems with Multi-Encoders and Multi-Decoders (2020.coling-main)

Challenge:	Pre-trained language models have led to substantial gains over a broad range of NLP tasks, but have limitations for high-quality tasks such as commonsense generation and ad keyword generation.
Approach:	They propose a Knowledge Filtering and Contrastive learning Network which references external knowledge and achieves better generation performance.
Outcome:	The proposed model outperforms the current state of the art on the CommonGen benchmark by a large margin.

Copied to clipboard

Yibin Shen, Cheqing Jin

Hidden Schema Networks (2023.acl-long)

Challenge:	Existing models that transform text descriptions into equation expressions only consider input/output objects as sequences, ignoring important structural information contained in text descriptions and equation expression.
Approach:	They propose a model that uses sequence-based encoders and graph-based decoders to enhance the representation of text descriptions and generate different equation expressions.
Outcome:	The proposed model outperforms existing state-of-the-art methods on a dataset with a n-word problem.

Copied to clipboard

Ramses Sanchez, Lukas Conrads, Pascal Welke, Kostadin Cvejoski, Cesar Ojeda Marin

Reasoning

Challenge:	Existing models that encode rich semantic and syntactic content are biased, but they are effective at encoding symbolic representations.
Approach:	They propose a neural language model that enforces explicit relational structures which allow for compositionality onto the output representations of pretrained language models.
Outcome:	The proposed model can encode sentences into sequences of symbols and infer the posterior distribution of the model from natural language datasets.

A Simple and Effective Approach to Automatic Post-Editing with Transfer Learning (P19-1)

Copied to clipboard

Gonçalo M. Correia, André F. T. Martins

Code Defect Detection Using Pre-trained Language Models with Encoder-Decoder via Line-Level Defect Localization (2024.lrec-main)

Challenge:	Existing APE systems generate artificial triplets of source sentences, machine translation outputs and human post-edits.
Approach:	They propose to use human post-edits to refine black-box machine translation (MT) models by fine-tuning pre-trained BERT models on both encoder and decoder of an APE system.
Outcome:	The proposed method improves on a dataset of 23K sentences on x86 GPUs.

Copied to clipboard

Jimin An, YunSeok Choi, Jee-Hyong Lee

Document-Level Neural Machine Translation with Hierarchical Attention Networks (D18-1)

Challenge:	Recent code Pre-trained Language Models (PLMs) have shown great success in code defect detection tasks.
Approach:	They propose a method that integrates line-level defect localization into a unified training process to identify which lines contain defects.
Outcome:	The proposed method significantly improves performance on four benchmark datasets for code defect detection.

Copied to clipboard

Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, James Henderson

SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation (2021.acl-long)

Challenge:	Neural machine translation (NMT) can be improved by including document-level contextual information.
Approach:	They propose a hierarchical attention model that captures document-level contextual information and conditioning on the NMT model’s own hidden states.
Outcome:	The proposed model improves the BLEU score over a strong NMT baseline with the state-of-the-art in context-aware methods and that both the encoder and decoder benefit from context in complementary ways.

Copied to clipboard

Shuo Ren, Long Zhou, Shujie Liu, Furu Wei, Ming Zhou, Shuai Ma

A Hierarchical Encoding-Decoding Scheme for Abstractive Multi-document Summarization (2023.findings-emnlp)

Challenge:	Using pre-training methods for NMT models is difficult because of the cross-attention module . cross-linguistic embeddings are not used to pretrain a decoder .
Approach:	They propose a semantic interface between pre-trained encoder and pre-train decoder to improve NMT performance.
Outcome:	The proposed method improves on translation and unsupervised translation tasks.

Copied to clipboard

Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, Lidong Bing

SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2022.acl-long)

Challenge:	Pre-trained language models have been used for abstractive single-document summarization (SDS) but they may not be suitable for multi-document summary (MDS)
Approach:	They propose to enforce hierarchy on both encoder and decoder to facilitate multi-document interactions for MDS.
Outcome:	Xiao et al. (2019) outperforms or is competitive with the previous best models.

Copied to clipboard

Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei

Multimodality

Challenge:	Existing work shows that pre-trained models can improve in various natural language processing tasks.
Approach:	They propose a unified-modal encoder-decoder framework that pre-trains speech-text representations using large-scale unlabeled speech and text data.
Outcome:	The proposed framework is superior to existing models on speech-to-text processing tasks.

Denoising based Sequence-to-Sequence Pre-training for Text Generation (D19-1)

Copied to clipboard

Liang Wang, Wei Zhao, Ruoyu Jia, Sujian Li, Jingming Liu

Language Branch Gated Multilingual Neural Machine Translation (2022.coling-1)

Challenge:	PoDA pre-trains encoders and decoders by denoising noise-corrupted text . Unlike encoder-only or decode-only methods, it can be used for text generation tasks without using any task-specific techniques.
Approach:	They propose a sequence-to-sequence (seq2sequ) pre-training method PoDA which denoises autoencoders by denoising noise-corrupted text.
Outcome:	The proposed method improves model performance over strong baselines without using any task-specific techniques and significantly speed up convergence.

Copied to clipboard

Haoran Sun, Deyi Xiong

Exploiting Deep Representations for Neural Machine Translation (D18-1)

Challenge:	Existing approaches to multilingual neural machine translation do not allow knowledge transfer across languages.
Approach:	They propose a language branch gated multilingual neural machine translation module that encourages knowledge transfer within the same language branch.
Outcome:	The proposed approach significantly improves translation quality on middle- and low-resource languages over previous methods.

Copied to clipboard

Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Shuming Shi, Tong Zhang

Improving Non-Autoregressive Neural Machine Translation via Modeling Localness (2022.coling-1)

Challenge:	Neural machine translation models typically implement encoder and decoder as multiple layers, but only the top layers are leveraged in the subsequent process, which misses the opportunity to exploit useful information embedded in other layers.
Approach:	They propose to expose all of these signals with layer aggregation and multi-layer attention mechanisms and introduce an auxiliary regularization term to encourage different layers to capture diverse information.
Outcome:	The proposed approach exposes all of these signals with layer aggregation and multi-layer attention mechanisms on widely-used translation datasets.

Copied to clipboard

Yong Wang, Xinwei Geng

Assist Non-native Viewers: Multimodal Cross-Lingual Summarization for How2 Videos (2022.emnlp-main)

Challenge:	Existing non-autoregressive neural machine translation models suffer from poor localization quality due to sequential dependencies within the target sentence.
Approach:	They propose to introduce local information into NAT models by explicitly introducing local information about surrounding words into the encoder and decoder sides to achieve localness-aware representations.
Outcome:	The proposed method can achieve significant improvements over strong NAT baselines.

Copied to clipboard

Nayu Liu, Kaiwen Wei, Xian Sun, Hongfeng Yu, Fanglong Yao, Li Jin, Guo Zhi, Guangluan Xu

Multimodality

Challenge:	Existing multimodal summarization methods are limited to monolingual videos . a proposed task aims to generate cross-lingual summaries from multimodal inputs .
Approach:	They propose a task to generate cross-lingual summaries from multimodal inputs of videos . they propose fusion network that integrates multimodal and cross-linguistic information .
Outcome:	The proposed task outperforms existing methods on a reorganized How2 dataset on the reorganized How2 data set.

Focus-Driven Contrastive Learning for Medical Question Summarization (2022.coling-1)

Copied to clipboard

Ming Zhang, Shuai Dou, Ziyang Wang, Yunfang Wu

A Multi-Task Incremental Learning Framework with Category Name Embedding for Aspect-Category Sentiment Analysis (2020.emnlp-main)

Challenge:	Existing methods to summarize health questions are not able to capture well question focus and lack the ability to understand sentence-level semantics.
Approach:	They propose a question focus-driven contrastive learning framework to capture question focus and exploit contrastive training at both encoder and decoder to obtain better sentence representations.
Outcome:	The proposed model achieves 5.33, 12.85 and 3.81 points over the baseline model on three medical benchmark datasets.

Copied to clipboard

Zehui Dai, Cheng Peng, Huajie Chen, Yadong Ding

Hard-Coded Gaussian Attention for Neural Machine Translation (2020.acl-main)

Challenge:	Sentiment analysis is an increasingly popular natural language processing task in academia and industry.
Approach:	They propose to use category name encoding network to weaken catastrophic forgetting problem . they set both encoder and decoder shared among all categories to weaker the catastrophic forgetting problem a .
Outcome:	The proposed model achieves state-of-the-art on two (T)ACSA benchmark datasets.

Copied to clipboard

Weiqiu You, Simeng Sun, Mohit Iyyer

Transforming Visual Scene Graphs to Image Captions (2023.acl-long)

Challenge:	Recent work has questioned the importance of multi-headed attention in achieving high translation quality.
Approach:	They develop a “hard-coded” attention variant without any learned parameters.
Outcome:	The proposed model reduces BLEU scores by adding a single learned cross attention head to an otherwise hard-coded Transformer.

Copied to clipboard

Xu Yang, Jiawei Peng, Zihua Wang, Haiyang Xu, Qinghao Ye, Chenliang Li, Songfang Huang, Fei Huang, Zhangzikang Li, Yu Zhang

Multimodality

Challenge:	Existing approaches to generate captions using image captioning are based on multi-head attention (MHA)
Approach:	They propose to transform scene graphs into more descriptive captions by using multi-head attention to build a Graph Neural Network (GNN) . they construct a Mixture-of-Expert (MOE)-based decoder where each expert is built on MHA for discriminating the graph embeddings to generate different kinds of words.
Outcome:	The proposed framework can generate captions from multiple visual features and objects . it is based on a mixture-of-expert (MOE)-based decoder based upon MHA .

Abstractive Text Summarization based on Language Model Conditioning and Locality Modeling (2020.lrec-1)

Copied to clipboard

Dmitrii Aksenov, Julian Moreno-Schneider, Peter Bourgonje, Robert Schwarzenberg, Leonhard Hennig, Georg Rehm